Graph Cut Algorithms for Binocular Stereo with Occlusions

Most binocular stereo algorithms assume that all scene elements are visible from both cameras. Scene elements that are visible from only one camera, known as occlusions, pose an important challenge for stereo. Occlusions are important for segmentation, because they appear near discontinuities. However, stereo algorithms tend to ignore occlusions because of their difficulty. One reason is that occlusions require the input images to be treated symmetrically, which complicates the problem formulation. Worse, certain depth maps imply physically impossible scene configurations, and must be excluded from the output. In this chapter we approach the problem of binocular stereo with occlusions from an energy minimization viewpoint. We begin by reviewing traditional stereo methods that do not handle occlusions. If occlusions are ignored, it is easy to formulate the stereo problem as a pixel labeling problem, which leads to an energy function that is common in early vision. This kind of energy function can be minimized using graph cuts, which is a combinatorial optimization technique that has proven to be very effective for low-level vision problems. Motivated by this, we have designed two graph cut stereo algorithms that are designed to handle occlusions. These algorithms produce promising experimental results on real data with ground truth. 1 Traditional stereo methods Computing stereo depth is a traditional problem in computer vision, and has been the focus of a great deal of work (see [5, 17] for recent surveys). Given a pair of images taken at the same time, two pixels are said to correspond if they show the same scene element. The goal of stereo is to compute correspondences between pixels, which then determines depth. The binocular stereo problem is typically formulated as follows: For every pixel in one image, find the corresponding pixel in the other image. We will refer to this as the traditional stereo problem. The problem formulation above has many advantages. It easily fits within a class of problems that arise in early vision called pixel labeling problems, 2 Vladimir Kolmogorov, Ramin Zabih where the goal is to assign each pixel p = (px, py) ∈ P a label from some set L. The label set L depends upon the particular problem; for example, in image denoising, L is intensities. In stereo, L consists of disparities. Pixel labeling problems have been widely studied in computer vision. The problem is naturally formulated in terms of energy minimization, where the goal is to find the labeling f = (f1, . . . , fp, . . . , f|P|) that minimizes